This invention relates generally to data storage, in particular, to digital data compression.
Modern large scale computing applications, such as social networking systems, receive and generate vast quantities of data continuously. Examples of data maintained by large scale applications include user information, images, videos, text posts, emails, performance logs, search indices, metadata, or any other suitable data. As the volume of data increases, the resources needed to store, serve, and secure this data increases correspondingly. Various data compression techniques may be to reduce the volume of data without losing information. Different data compression technique make tradeoffs between factors such as compression speed, decompression speed, compression efficiency, and memory usage during compression. Compression speed is determined by the complexity of the steps for compressing data. Similarly, decompression speed is determined by the complexity of the steps for decompressing data. Compression efficiency measures the size of the compressed data relative to the uncompressed data.
Various requirements determine the compression technique used for a system. For example, if data is stored offline compression speed is immaterial, a compression technique with high compression efficiency and high decompression speed may be selected at the expense of compression speed. For example, a system storing and streaming movies may use a compression technique having poor compression speed but good compression efficiency and decompression speed. On the other hand, for systems storing and serving data in real-time—websites, social networking systems, games, etc.—compression speed and decompression speed may be more important than compression efficiency.